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An  important  and  continuing  concern  within  the  Army  Research  Insti- 
tute for  the  Behavioral  and  Social  Sciences  (ARI)  has  been  the  influence 
of  factors  that  may  enhance  or  degrade  the  usefulness  of  responses  to 
questionnaire  and  interview  items.  Examples  of  such  factors  are  the  num- 
ber of  items  in  the  questionnaire,  the  number  of  response  alternatives  in 
items,  the  presence  of  words  in  item  stems  that  may  bias  the  direction  or 
intensity  of  responses,  the  kind  of  modifiers  used  in  response  alterna- 
tives, the  forcing  or  not  forcing  of  the  respondent  to  state  a preference, 
the  serial  order  of  items  and  response  alternatives,  and  the  amount  of 
time  required  to  complete  the  questionnaire. 

This  report  represents  a collaborative  ARI  effort  designed  to  pro- 
vide objective  data  concerning  the  denotative  meanings  and  inherent  scale 
positions  of  common  descriptors  used  as  response  alternatives  and  scale 
anchors  in  questionnaire  construction.  The  statistical  analyses  were  con- 
ducted by  Operations  Research  Associates,  Palo  Alto,  Calif.,  in  connec- 
tion with  a contract  that  subsumed  their  contribution  to  the  present  re- 
port (Contract  No.  DAHC  19-74-C-0032 ) . The  larger  project  was  directed 
at  producing  an  instructional  manual  for  constructing  and  administering 
questionnaires  (ARI  Special  Publication  P-77-1)  and  a technical  report  of 
the  results  of  arT- ex£'ensive'_survey  of  the  literature  on  factors  involved 
in  questionnaire  construction  and  administration.  Data  collection  and 
manuscript  preparation  of  this  report  were  accomplished  by  the  ARI  Field 
Unit,  Fort  Hood,  Tex. 

This  report  supports  the  objectives  of  FY  1974  RDTE  Project 
2Q763743A775 , Human  Performance  in  Field  Assessment. 


JCfSEPH  Z&PDNER 

echnical  Director  (Designate) 


THE  PERCEIVED  FAVORABLENESS  OF  SELECTED  SCALE  ANCHORS  AND  RESPONSE 
ALTERNATIVES 


BRIEF 


Requirement: 

\ 

In  the  interest  of  improving  methods  of  questionnaire  construction, 
the  principal  purpose  of  this  study  was  to  produce  objective  norms  regard- 
ing the  extent  to  which  respondents  perceive  various  descriptive  phrases 
(denoting  degrees  of  acceptability,  adequacy,  or  goodness)  as  indicators 
of  favorable  or  unfavorable  attitude. 


Procedure : 

Lists  containing  many  descriptors  denoting  degrees  of  acceptability, 
adequacy,  and  goodness  were  distributed  to  a random  selection  of  5 1 Army 
officers  and  enlisted  personnel  who  rated  each  descriptor  on  a bipolar 
scale  of  favorableness  and  indicated  for  each  whether  it  seemed  ambiguous. 


Findings : 

The  majority  of  the  tested  descriptors  were  considered  unambiguous  by 
a majority  of  the  raters.  In  this  connection,  a specific  finding  of  inter- 
est was  that  the  adverb  modifiers  pretty  and  rather  were  contained  in  more 
than  half  the  adjective  phrases  considered  ambiguous  by  the  raters,  which 
indicated  the  advisability  of  exercising  caution  in  using  these  terms  in 
questionnaire  response  alternatives  and  scale  anchors. 

The  mean,  standard  deviation,  and  range  of  the  raters'  responses  for 
each  descriptor  were  computed  and  are  presented  in  tabular  form  for  use  in 
questionnaire  construction.  Descriptors  with  large  standard  deviations 
are  noted  and  considered  generally  less  desirable  than  the  others  but  are 
not  necessarily  to  be  avoided  in  every  application. 

A limited  amount  of  data  relevant  to  intrasubject  consistency  indi- 
cated that  the  ratings  are  reliable. 

Examples  of  sets  of  scaled  response  descriptors  with  desirable  scale 
properties  are  presented  in  connection  with  a description  of  possible  meth- 
ods for  choosing  response  descriptors  for  questionnaire  items. 

Regarding  scale  midpoints,  it  was  noted  that  the  term  neutral,  al- 
though possessing  desirable  statistical  characteristics,  may  normally  be 
unsuitable  as  a midpoint  descriptor  because  of  an  inherent  ambiguity  of 
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meaning.  The  term  borderline  seems  appropriate  as  a replacement  for  neu- 
tral in  many  contexts. 

Utilization  of  Findings: 

The  objective  contents  of  this  report  have  been  integrated  into  the 
Army  Research  Institute  Questionnaire  Construction  Manual,  which  elabo- 
rates procedures  for  selecting  response  alternatives  and  scale  anchors 
(and  other  topics).  The  manual  is  used  within  both  the  Army  and  the  ci- 
vilian community.  This  report  also  complements  existing  literature  per- 
taining to  the  characteristics  of  descriptive  terms  for  use  in  question- 
naire item  construction. 
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THE  PERCEIVED  FAVORABLENESS  OF  SELECTED  SCALE  ANCHORS 
AND  RESPONSE  ALTERNATIVES 


INTRODUCTION 


The  construction  of  multiple-choice  rating-scale  questionnaires  fre- 
quently involves  selecting  adjectives,  adverbs,  or  descriptive  phrases 
for  use  as  response  alternatives  or  scale  anchors.  The  ultimate  value 
of  the  questionnaire  often  rests  to  a great  degree  on  such  properties  of 
these  descriptors  as  clarity  of  meaning,  evaluative  intensity,  and  the 
extent  to  which  the  denotations  of  the  descriptors  do  not  overlap.  It  is 
desirable,  therefore,  that  response  scales  be  constructed  with  such  fac- 
tors in  mind  and  in  accordance  with  the  objective  properties  of  the  candi- 
date descriptors. 

Objective  information  about  the  characteristics  of  candidate  descrip- 
tors is  available  from  several  studies  reported  in  the  psychological  lit- 
erature (e.g. , Altemeyer,  1970;  Cliff,  1959;  Dodd  & Gerberick,  1960; 
Gividen,  1973;  Jones  & Thurstone,  1955;  Myers  & Warner,  1968;  Mosie ; , 

1941;  and  U.S.  Army  Test  and  Evaluation  Command,  1973).  Much  of  the  rel- 
evant data  in  the  literature  (including  the  objective  contents  of  this 
report)  (Matthews,  Wright,  & Yudowitch,  1975)  has  been  abstracted  or 
cited  in  a comprehensive  literature  survey  prepared  for  the  Army  Research 
Institute  (ARI)  by  Dyer,  Matthews,  Stulac,  Wright,  and  Yudowitch  (1976) 
and  integrated  into  the  ARI  Questionnaire  Construction  Manual  (Dyer, 
Matthews,  Wright,  Yudowitch,  & Nystrom,  1976). 

The  primary  purpose  of  this  ARI  study  was  to  complement  previously 
published  studies  dealing  with  the  characteristics  of  response  alterna- 
tives and  scale  anchors.  More  specifically,  it  was  desirable  for  ARI  to 
obtain  norms,  based  on  military  respondents,  regarding  the  perceived  char- 
acteristics of  descriptors  that  denote  degrees  of  acceptability,  adequacy, 
and  relative  goodness.  The  following  are  examples  of  these  three  types  of 
descriptors. 

Acceptability  Adequacy  Goodness 

descriptors  descriptors descriptors 

Wholly  acceptable  Extremely  inadequate  Best  of  all 

Slightly  acceptable  Somewhat  inadequate  Alike 

Highly  unacceptable  Completely  inadequate  Absolutely  worst 

Of  primary  concern  in  this  study  was  the  extent  to  which  such  de- 
scriptors are  perceived  by  respondents  as  signifying  favorable  or  unfavor- 
able (positive  or  negative)  attitude.  Thus,  the  primary  procedural  objec- 
tive of  the  study  was  to  determine  for  each  of  a sizable  selection  of  the 
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A secondary  purpose  associated  with  this  study  was  to  demonstrate  the 
construction  of  response  alternative  sets  from  normative  data. 


PROCEDURES 


Pilot  Test 


An  original  pool  of  220  descriptors  representing  the  three  aforemen- 
tioned categories  was  prepared  by  Operations  Research  Associates  (ORA). 
Individual  items  were  garnered  from  the  literature  or  created  especially 
for  this  study.  The  intent  was  to  generate  a list  for  each  category  that 
would  contain  most  of  the  descriptors  of  that  type  that  researchers  might 
use  in  the  construction  of  questionnaires.  The  220  items  contained  62  ac- 
ceptability descriptors,  62  adequacy  descriptors,  and  96  relative  good- 
ness descriptors. 

The  three  original  lists  were  pilot  tested  by  ARI , using  10  Army  of- 
ficers as  respondents.  Each  respondent  was  asked  to  rate  each  of  the  220 
descriptors  on  an  1 1 -point  favorableness  scale  ranging  from  1 to  11,  with 
6 as  the  neutral  point.  The  respondents  read  the  following  instructions 
prior  to  rating. 

Given  on  the  following  pages  are  three  lists  of  phrases.  To 
the  right  of  each  phrase  is  an  11-point  scale  ranging  from  1 
to  11  on  which  1 means  Most  Unfavorable,  6 means  Neither  Un- 
favorable nor  Favorable,  and  11  means  Most  Favorable.  The 
distance  between  each  number  should  be  regarded  as  equal. 

Indicate  by  circling  one  number  for  each  phrase  the  degree 
of  favorableness  expressed  by  that  phrase.  Cross  out  any 
phrase  the  meaning  of  which  is  unclear  to  you. 

Following  the  rating  procedure,  comments  regarding  the  items  and  rating 
procedure  were  solicited  from  the  respondents. 

The  data  from  the  rating  procedure  were  analyzed  by  calculating  the 
standard  deviation  of  the  10  (or  fewer)  responses  for  each  of  the  220  de- 
scriptors. In  addition,  the  number  of  times  each  descriptor  was  crossed 
out  (rejected  as  unclear  or  unratable)  was  recorded. 

Subsequent  to  the  analysis,  the  three  lists  were  edited  by  elimina- 
ting each  descriptor  whose  standard  deviation  was  1.00  or  greater  or  that 
was  rejected  as  ambiguous  by  two  or  more  of  the  respondents. 

The  comments  of  the  respondents  indicated  that  the  third  list  of 
descriptors — the  relative  goodness  items — contained  two  general  types  of 
items,  some  expressing  degrees  of  likeness,  or  equality,  others  express- 
ing degrees  of  relative  goodness.  Furthermore,  the  respondents  reported 
having  experienced  difficulty  in  attempting  to  rate  the  former  type  on 
the  favorableness  scale.  Consequently,  the  list  was  further  edited  to 
eliminate  items  of  the  likeness  type  that  caused  respondents  confusion. 
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The  editing  yielded  48  acceptability,  49  adequacy,  and  41  relative 
goodness  descriptors. 

The  comments  of  the  respondents  also  indicated  that  the  favorableness 
scale  would  probably  be  more  compatible  with  the  required  rating  task  if 
the  scale  values  ranged  from  -5  (most  unfavorable)  to  + 5 (most  favorable), 
with  0 as  neutral,  instead  of  from  1 to  11. 

Main  Test 

I 

Test  Lists.  Two  of  the  descriptors  in  the  acceptability  list  (mildly 
acceptable  and  highly  unacceptable)  were  included  twice,  creating  a 50- 
item  list.  One  descriptor  in  the  adequacy  list  (exceptionally  inadequate) 
was  included  twice,  also  creating  a 50-item  list.  No  repetitions  were  in- 
corporated into  the  relative  goodness  list,  so  the  final  test  version  con- 
tained only  the  41  items  that  resulted  from  the  pilot -test  revisions.  In 
addition  to  the  repetitions  within  lists,  there  were  certain  repetitions 
across  lists:  The  descriptors  neutral  and  borderline  occurred  in  all 
three  lists;  the  descriptor  marginal  occurred  in  the  acceptability  and 
relative  goodness  lists  but  was  inadvertently  left  off  the  adequacy  lists. 
These  repetitions  within  and  across  lists  permitted  a crude  assessment  of 
intrasubject  response  consistency. 

■ Each  list  was  prepared  so  that  the  descriptors  were  presented  in  a 

haphazard  serial  order. 

Administration  of  Test  Lists.  Questionnaires  containing  the  three 
lists  were  distributed  to  50  Army  officers  and  50  Army  enlisted  men  as- 
signed to  the  Modern  Army  Selected  Systems  Test,  Evaluation,  and  Review 
(MASSTER),  Fort  Hood,  Tex.  These  subjects  were  randomly  sampled  from 
the  MASSTER  telephone  directory.  A cover  letter  accompanying  the  ques- 
tionnaire included  the  following  statement. 

A significant  portion  of  the  data  collected  in  MASSTER  tests 
is  subjective  and  judgmental  in  nature.  Collection  of  this 
type  of  data  frequently  requires  construction  of  question- 
naires. In  order  to  generate  a list  of  response  alternatives 
which  have  been  evaluated  in  terms  of  their  meanings  to  re- 
spondents, your  cooperation  is  needed  in  evaluating  the  en- 
closed phrase  lists.  Please  return  the  completed  phrase  list 
to  the  ARI  Field  Unit  .... 

The  questionnaire  itself  contained  the  same  instructions  as  those  used  in 
the  pilot  test  (see  above),  except  that  the  favorableness  scale  was  de- 
scribed as  ranging  from  -5  to  +5,  with  a midpoint  of  0. 

Completed  questionnaires  were  returned  by  92  of  the  100  subjects. 
Unfortunately,  however,  60  of  the  returned  questionnaires  had  been  com- 
pleted under  an  apparent  misunderstanding  about  the  intent  of  the  instruc- 
tions. The  respondents  involved  obviously  had  not  rated  the  candidate 
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descriptors  on  the  extent  to  which  they  were  perceived  as  signifying  fa- 
vorable or  unfavorable  attitudes.  Rather,  the  descriptors  appeared  to 
have  been  assessed  on  the  basis  of  the  respondents'  personal  preferences 
for  them.  Therefore,  questionnaires  were  redistributed  to  those  respond- 
ents and  to  the  few  respondents  who  had  not  returned  questionnaires  from 
the  first  distribution. 

The  cover  letter  accompanying  the  second  questionnaire  contained  state- 
ments intended  to  clear  up  the  misunderstanding.  They  read  as  follows. 
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As  part  of  its  support  to  MASSTER,  ARI  and  a contractor  are 
investigating  various  factors  which  influence  obtaining  judg- 
ments and  subjective  evaluations  of  equipment,  procedures  and 
concepts  by  participants/observers  in  MASSTER  tests. 

Attached  . . . are  three  lists  of  phrases  which  were  sent  . . . 
earlier  ....  The  same  lists  of  phrases  are  being  sent  out 
again  with  additional  guidance  because  there  was  apparently  a 
misunderstanding  by  many  respondents  concerning  the  task  in- 
volved. Many  of  the  phrase  lists  were  returned  with  the 
phrases  rated  according  to  whether  the  respondent  approved 
or  disapproved  of  the  phrase  as  a descriptor — while  what  was 
desired  was  their  judgment  of  how  positive/favorable  or  nega- 
tive/unfavorable each  rating  phrase  would  be  if  it  were  given 
as  a rating  of  some  equipment,  procedure  or  concept.  Please 
process  the  attached  lists  of  rating  phrases  in  accordance 
with  the  guidance  below  and  return  to  the  ARI  Field  Unit  . . . . 

Guidance.  List  A consists  of  phrases  used  to  express  dif- 
ferent degrees  of  adequacy  or  inadequacy.  List  B consists 
of  phrases  used  to  express  different  degrees  of  acceptability 
or  unacceptability.  List  C consists  of  phrases  used  to  com- 
pare an  item  with  a standard  in  terms  of  better  or  worse. 
Opposite  each  rating  phrase  are  eleven  numbers,  -5  to  +5, 
which  constitute  a scale  for  you  to  use  to  tell  us  how  nega- 
tive/unfavorable or  positive/favorable  each  rating  phrase  would 
be  for  you  if  you  used  it  to  rate  some  equipment,  procedure  or 
concept.  A rating  of  -5  stands  for  the  most  negative  or  un- 
favorable rating  possible,  and  +5  stands  for  the  most  positive 
or  favorable  rating  possible,  with  less  extreme  degrees  of 
favorableness  or  unfavorableness  represented  by  the  numbers 
in  between.  A rating  of  0 means  that  the  phrase  is  neither 
negative/unfavorable  nor  positive/favorable.  For  each  rating 
phrase,  circle  the  number  which  describes  how  negative/unfavor- 
able or  positive/favorable  the  rating  phrase  is  when  applied 
to  you.  For  example,  if  you  saw  in  a list  the  phrase  highly 
satisfactory  and  you  considered  it  to  be  a very  positive/favor- 
able descriptor,  you  would  record  your  response  by  circling  a 
+4  or  +5  as  shown  below. 

Highly  satisfactory  -5  -4  -3  -2  -1  0 +1  +2  +3  ^4^+5 
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On  the  other  hand,  if  you  saw  the  phrase  highly  unsatsifactory , 
you  might  give  it  a -4  or  -5  as  shown  below  since  it  is  an 
unfavorable/negative  descriptor. 

Highly  unsatisfactory  -5  <3  -3  -2  -1  0 +1  +2  +3  +4  +5 

If  you  saw  the  phrase  neither  satisfactory  or  unsatisfactory, 
you  might  consider  it  to  be  a neutral  descriptor  and  give  it  a 
rating  of  0 on  the  11-point  scale. 

Note  that  the  dimension  on  which  the  phrases  are  to  be  rated 
is  their  meaning — not  your  personal  approval  or  disapproval 
of  the  phrases  as  descriptors. 

If  you  have  any  questions  concerning  these  rating  phrase  lists 
or  how  to  process  them  please  call  Dr.  [name]  at  [phone  number]. 


RESULTS 

Sample  Characteristics 

The  number  of  completed  questionnaires  obtained  from  the  second  dis- 
tribution was  22.  Two  of  these  were  eliminated  from  the  analysis  because 
more  than  half  the  items  were  crossed  out  by  the  respondents.  Another 
was  eliminated  because  more  than  10%  of  the  responses  were  judged  unreal- 
istic (e.g. , completely  adequate  was  rated  -5,  and  extremely  inadequate 
was  rated  +5).  Thus  the  second  distribution  yielded  19  valid  question- 
naires. These,  combined  with  the  32  valid  questionnaires  from  the  first 
distribution,  produced  a total  of  51  usable  questionnaires. 

Specific  items  on  some  of  these  questionnaires  were  eliminated  from 
the  analyses  because  the  response  was  considered  very  unreasonable,  more 
than  one  scale  value  had  been  circled  by  the  respondent,  or  the  item  had 
been  crossed  out  as  ambiguous  by  the  respondent.  Table  1 shows  the  candi- 
date descriptors  that  were  crossed  out  by  two  or  more  respondents.  Four- 
teen of  the  141  descriptors  were  crossed  out,  but  never  by  more  than  4 of 
the  51  respondents.  It  appears,  then,  that  most  of  the  descriptors  were 
sufficiently  unambiguous  to  most  of  the  respondents.  It  should  also  be 
noted  that  6 of  the  crossed-out  descriptors  contained  the  adverb  modifier 
pretty  or  rather. 


Normative  Attributes  of  the  Candidate  Descriptors 

Tables  2,  3,  and  4,  respectively,  portray  the  acceptability,  adequacy, 
and  relative  goodness  lists  and  the  results  of  the  primary  analyses.  In 
each  table  the  first  four  columns  of  figures  describe  the  distribution  of 
responses  for  each  candidate  descriptor.  Consider,  as  an  example,  the 
first  row  of  Table  2:  the  mean  4.73  indicates  that,  on  the  average,  the 
subjects  felt  that  wholly  acceptable  denoted  a highly  positive  attitude  or 
degree  of  favorableness  (recall  that  the  favorableness  scale  extended  from 
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Table  1 
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CANDIDATE  DESCRIPTORS  WHOSE  MEANINGS  WERE 
UNCLEAR  TO  TWO  OR  MORE  RESPONDENTS 


Descriptor 

No.  of  respondents 

"Acceptability"  descriptors: 

Pretty  acceptable 

2 

Reasonably  unacceptable 

2 

"Adequacy"  descriptors: 

Considerably  inadequate 

3 

Exceptionally  adequate 

2 

Mildly  adequate 

2 

Pretty  adequate 

2 

Pretty  inadequate 

2 

Rather  adequate 

2 

Rather  inadequate 

2 

Somewhat  adequate 

2 

"Relative  goodness"  descriptors: 

Rather  better 

2 

Descriptors  for  scale  midpoints: 

Borderl ine 

4 

Marginal 

3 

Neutral 

2 

Note.  Descriptors  within  each  category  are  listed  alphabetically. 
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Table  2 


RESULTS  PERTAINING  TO  "ACCEPTABILITY"  DESCRIPTORS 


Descriptor 

Mean 

SD 

Range 

No.  of 

Min.  Max. 

subjects 

Wholly  acceptable 

4.73 

.56 

3 

5 

51 

Completely  acceptable 

4.69 

.61 

3 

5 

51 

Fully  acceptable 

4.41 

.87 

2 

5 

51 

Extremely  acceptable 

4.39 

.72 

3 

5 

51 

Most  acceptable 

4.16 

.92 

2 

5 

51 

Very  very  acceptable 

4.16 

.83 

2 

5 

51 

Highly  acceptable 

4.04 

.63 

3 

5 

50 

Quite  acceptable 

3.22 

.96 

1 

5 

51 

Largely  acceptable 

3.14 

.99 

1 

5 

51 

Acceptable 

2.39 

1 .46 

0 

5 

51 

Reasonably  acceptable 

2.29 

.72 

1 

4 

51 

Moderately  acceptable 

2.28 

.72 

1 

3 

50 

Pretty  acceptable 

2.00 

1.13 

-3 

4 

49 

Rather  acceptable 

1.94 

.82 

0 

4 

49 

Fairly  acceptable 

1.84 

.92 

0 

4 

50 

Mildly  acceptable 

1.80 

.95 

-1 

4 

51 

Mildly  acceptable 

1.69 

.70 

-1 

4 

51 

Somewhat  acceptable 

1.46 

1.24 

-2 

3 

48 

Barely  acceptable 

1.08 

.52 

-1 

3 

51 

Slightly  acceptable 

1.04 

.52 

-1 

2 

51 

Sort  of  acceptable 

.94 

.65 

-1 

2 

50 

Borderline 

.00 

.20 

-1 

1 

50 

Neutral 

.00 

.00 

0 

0 

51 

Marginal 

- .12 

.52 

-2 

1 

50 

Barely  unacceptable 

-1.10 

.30 

-2 

-1 

50 

Slightly  unacceptable 

-1 .26 

.59 

-4 

-1 

51 

Somewhat  unacceptable 

-1 .77 

.67 

-3 

-1 

51 

Rather  unacceptable 

-2.02 

.84 

-4 

0 

50 

Fairly  unacceptable 

-2.16 

.88 

-5 

-1 

50 

Moderately  unacceptable 

-2.34 

.68 

-3 

-1 

50 

Pretty  unacceptable 

-2.41 

.66 

-4 

-1 

51 

Reasonably  unacceptable 

-2.44 

.75 

-4 

-1 

50 

Unacceptable 

-2.67 

1.38 

-5 

0 

51 

Substantially  unacceptable 

-3.24 

.90 

-5 

-1 

51 

(Continued) 
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Table  2 (cont) 


Descriptor 

Mean 

SD 

Range 

Min.  Max. 

No.  of 
subjects 

Quite  unacceptable 

-3.39 

1.07 

-5 

0 

49 

Largely  unacceptable 

-3.39 

.82 

-5 

-1 

51 

Considerably  unacceptable 

-3.44 

.78 

-5 

-2 

50 

Notably  unacceptable 

-3.50 

1.04 

-5 

-1 

50 

Decidedly  unacceptable 

-3.84 

1.02 

-5 

-1 

49 

Highly  unacceptablea 

-4.22 

.58 

-5 

-3 

50 

Highly  unacceptableb 

-4.29 

.54 

-5 

-3 

51 

Most  unacceptable 

-4.42 

.72 

-5 

-2 

50 

Very  very  unacceptable 

-4.49 

.50 

-5 

-4 

51 

Exceptionally  unacceptable 

-4.54 

.61 

-5 

-3 

50 

Extremely  unacceptable 

-4.69 

.46 

-5 

-4 

51 

Completely  unacceptable 

-4.90 

.36 

-5 

-3 

.50 

Entirely  unacceptable 

-4.90 

.36 

-5 

-3 

50 

Wholly  unacceptable 

-4.92 

.27 

-5 

-4 

51 

Absolutely  unacceptable 

-4.92 

.33 

-5 

-3 

51 

Totally  unacceptable 

-4.94 

.24 

-5 

-4 

51 

aFirst  usage  in  test  list. 
bSecond  usage  in  test  list. 
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Table  3 


RESULTS  PERTAINING  TO  "ADEQUACY"  DESCRIPTORS 


Descriptor 

Mean 

SD 

Range 

Min.  Max. 

No.  of 
subjects 

Totally  adequate 

4.62 

.85 

2 

5 

50 

Absolutely  adequate 

4.54 

.92 

1 

5 

50 

Completely  adequate 

4.49 

.83 

2 

5 

51 

Extremely  adequate 

4.41 

.72 

3 

5 

51 

Exceptionally  adequate 

4.38 

.87 

2 

5 

50 

Entirely  adequate 

4.34 

.86 

2 

5 

50 

Wholly  adequate 

4.31 

1.04 

1 

5 

51 

Fully  adequate 

4.29 

.91 

2 

5 

51 

Very  very  adequate 

4.06 

.88 

1 

5 

48 

Perfectly  adequate 

3.92 

1 .03 

2 

5 

51 

Highly  adequate 

3.84 

.61 

2 

5 

51 

Most  adequate 

3.84 

.98 

2 

5 

51 

Very  adequate 

3.42 

.85 

1 

5 

50 

Decidedly  adequate 

3.14 

1.54 

-4 

5 

50 

Considerably  adequate 

3.02 

.87 

1 

5 

51 

Quite  adequate 

2.98 

.98 

1 

5 

49 

Largely  adequate 

2.86 

.99 

-1 

4 

51 

Substantially  adequate 

2.61 

1 .03 

-1 

5 

51 

Reasonably  adequate 

2.41 

.77 

1 

4 

51 

Pretty  adequate 

2.31 

.86 

0 

5 

49 

Rather  adequate 

1.76 

.89 

0 

4 

49 

Mildly  adequate 

1.57 

.67 

1 

3 

49 

Somewhat  adequate 

1.32 

.79 

-2 

3 

49 

Slightly  adequate 

1.20 

.57 

-1 

3 

50 

Barely  adequate 

.63 

.93 

-4 

2 

51 

Neutral 

.00 

.00 

0 

0 

50 

Borderline 

- .02 

.32 

-1 

1 

50 

Barely  inadequate 

-1.16 

.64 

-4 

0 

51 

Mildly  inadequate 

-1.35 

.62 

-3 

0 

51 

Slightly  inadequate 

-1.38 

.77 

-4 

1 

50 

Somewhat  inadequate 

-1 .88 

.73 

-3 

0 

51 

Rather  inadequate 

-2.10 

.97 

-4 

0 

49 

Moderately  inadequate 

-2.16 

1.02 

-4 

2 

51 

Fairly  inadequate 

-2.22 

.80 

-4 

1 

51 

Pretty  inadequate 

-2.35 

.96 

-4 

1 

49 

Considerably  inadequate 

-3.60 

.68 

-5 

-2 

45 

(Continued) 


Table  3 (cont) 


Descriptor 

Mean 

SD 

Range 

Min.  Max. 

No.  of 
subjects 

Very  inadequate 

-3.74 

.78 

-5 

-2 

49 

Decidedly  inadequate 

-3.78 

.94 

-5 

-1 

50 

Most  inadequate 

-3.98 

1.55 

-5 

4 

49 

Highly  inadequate 

-4.20 

.74 

-5 

-3 

51 

Very  very  inadequate 

-4.46 

.54 

-5 

-3 

50 

Exceptionally  inadequateb 

-4.56 

.64 

-5 

-3 

50 

Extremely  inadequate 

-4.60 

.53 

-5 

-3 

51 

Fully  inadequate 

-4.67 

.68 

-5 

-2 

51 

Exceptionally  inadequate9 

-4.68 

.51 

-5 

-3 

50 

Wholly  inadequate 

-4.78 

.50 

-5 

-3 

51 

Entirely  inadequate 

-4.79 

.64 

-5 

-2 

48 

Completely  inadequate 

-4.80 

.53 

-5 

-3 

50 

Absolutely  inadequate 

-4.88 

.43 

-5 

-3 

50 

Totally  inadequate 

-4.90 

.41 

-5 

-3 

50 

aFirst  usage  in  test  list. 
bSecond  usage  in  test  list. 


Table  4 


RESULTS  PERTAINING  TO  "RELATIVE 

GOODNESS 

" DESCRIPTORS 

Descriptor 

Mean 

SD 

Range 

Min.  Max. 

No.  of 
subjects 

Best  of  all 

4.90 

.51 

2 

5 

48 

Absolutely  best 

4.84 

.46 

3 

5 

51 

Truly  best 

4.60 

.72 

3 

5 

50 

Undoubtedly  best 

4.57 

.82 

2 

5 

51 

Decidedly  best 

4.37 

.84 

2 

5 

51 

Best 

4.22 

1.46 

-3 

5 

51 

Absolutely  better 

4.06 

.99 

2 

5 

50 

Extremely  better 

3.92 

.88 

2 

5 

51 

Substantially  better 

3.70 

.92 

2 

5 

50 

Decidedly  better 

3.41 

.93 

1 

5 

51 

Conspicuously  better 

3.06 

.80 

2 

5 

51 

Moderately  better 

2.26 

.74 

1 

4 

51 

Somewhat  better 

1 .84 

.80 

1 

4 

51 

Rather  better 

1 .82 

.72 

1 

4 

49 

Slightly  better 

1.16 

.78 

-1 

4 

51 

Barely  better 

.96 

.66 

-1 

3 

51 

Absolutely  alike 

.59 

1.62 

-1 

5 

51 

Alike 

.22 

.85 

0 

5 

51 

The  same 

.16 

.80 

0 

5 

51 

Neutral 

.00 

.00 

0 

0 

50 

Borderl ine 

- .06 

.31 

-1 

1 

49 

Marginal 

- .18 

.92 

-4 

2 

49 

Barely  worse 

-1 .04 

.82 

-3 

4 

51 

SI ightly  worse 

-1.22 

.50 

-3 

-1 

51 

Somewhat  worse 

-2.08 

.86 

-4 

-1 

51 

Moderately  worse 

-2.22 

.94 

-4 

2 

50 

Noticeably  worse 

-2.53 

1 .04 

-4 

2 

51 

Worse 

-2.67 

1.42 

-5 

-1 

51 

Notably  worse 

-3.02 

1.04 

-5 

-1 

51 

Largely  worse 

-3.22 

1.11 

-5 

3 

51 

Considerably  worse 

-3.28 

. 1.21 

-5 

4 

51 

Conspicuously  worse 

-3.28 

.89 

-5 

-1 

51 

Much  worse 

-3.29 

.81 

-5 

-1 

49 

Substantially  worse 

-3.46 

.90 

-5 

-2 

50 

(Continued) 
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Table  4 (cont) 


Descriptor 

Mean 

SD 

Range 

Min.  Max. 

No.  of 
subjects 

Decidedly  worse 

-3.76 

.91 

-5 

-2 

50 

Very  much  worse 

-3.94 

.75 

-5 

-2 

51 

Absolutely  worse 

-4.43 

.82 

-5 

-2 

51 

Decidedly  worst 

-4.43 

.75 

-5 

-2 

51 

Undoubtedly  worst 

-4.51 

.87 

-5 

-2 

51 

Absolutely  worst 

-4.69 

1.29 

-5 

4 

51 

Worst  of  all 

-4.78 

1.30 

-5 

4 

49 

12 


-5  to  +5);  the  standard  deviation  (SD)  .56  indicates  that  differences  among 
the  subjects'  responses  for  this  item  were,  on  the  average,  a little  more 
than  one-half  scale  unit;  the  range  columns  give  the  smallest  and  the  larg- 
est scale  value  given  to  this  item  by  any  subject.  The  last  column  in  each 
table  shows  the  number  of  subjects  whose  responses  were  analyzed  for  each 
candidate  descriptor. 

Each  of  the  means  given  in  Tables  2,  3,  and  4 is,  in  effect,  a com- 
promise among  different  estimates  of  many  respondents.  The  standard  de- 
viations, on  the  other  hand,  measure  the  degree  of  consensus  among  the 
respondents.  Thus,  the  standard  deviations  permit  examination  of  the 
"clarity"  of  the  descriptors  in  a manner  different  from  that  used  to  de- 
rive Table  1.  Table  1 shows  the  descriptors  that  were  considered  ambigu- 
ous by  the  respondents.  In  contrast,  the  standard  deviations  in  Tables  2, 
3,  and  4 show  whether  the  respondents  agreed  on  the  scale  values  on  those 
descriptors  that  they  considered  unambiguous.  Descriptors  with  large 
standard  deviations  possess  favorableness  values  that  vary  considerably 
from  subject  to  subject  and,  therefore,  would  often  be  less  desirable  as 
point  anchors  for  rating  scales. 

Table  5 lists  all  candidate  descriptors  with  standard  deviations 
greater  than  1.25  (an  arbitrary  cutoff).  Note  that  for  the  most  part,  the 
terms  listed  seem  to  be  characterized  by  extremeness  or  generality. 

Table  5 

CANDIDATE  DESCRIPTORS  WITH  STANDARD  DEVIATIONS  GREATER  THAN  1.25 


Descriptor 

SD 

Acceptable 

1 .46 

Unacceptable 

1.38 

Most  inadequate 

1.55 

Decidedly  adequate 

1.54 

Absolutely  al ike 

1.62 

Best 

1 .46 

Worse 

1.42 

Worst  of  all 

1.30 

Absolutely  worst 

1.29 

Note.  The  descriptors  within  subgroups  are  listed  in  order  of 
decreasing  SD. 
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Intrasubject  Consistency 


The  results  pertaining  to  descriptors  that  were  repeated  either  with- 
in or  across  test  lists  have  been  gathered  together  in  Table  6.  The  table 
shows  that  the  two  means  for  each  of  the  three  descriptors  repeated  within 
lists  were  quite  similar,  considering  the  1 1 -point  range  of  the  favorable- 
ness scale.  The  same  holds  true  for  descriptors  repeated  across  lists. 

The  standard  deviations  reveal  a similar  pattern.  The  variabilities  asso- 
ciated with  repetitions  of  a given  descriptor  appear  quite  similar — the 
largest  discrepancy  is  .40,  between  the  two  repetitions  of  marginal.  Thus 
the  subjects'  responses  seemed  to  exhibit  a satisfactory  degree  of  consis- 
tency with  respect  to  the  few  items  repeated  within  or  across  lists.  Be- 
cause of  the  limited  data  available  regarding  intrasubject  consistency, 
further  analyses  were  not  conducted. 


Example  Sets  of  Minimally  Overlapping  Response  Descriptors 

The  construction  of  good  questionnaire  items  of  the  type  under  con- 
sideration requires  that  the  denotations  of  response  alternatives  or  scale 
anchors  located  at  different  points  on  the  response  scale  do  n^t  overlap 
to  an  extent  that  might  cause  respondents  to  attribute  the  same  meaning  to 
any  two  different  points  on  the  scale.  Hence  it  is  often  desirable  to 
choose  descriptors  that  occupy  narrow  bands  along  the  response  dimension. 
In  this  section,  several  examples  of  lists  of  minimally  overlapping  de- 
scriptors are  presented. 

Nonoverlapping  or  minimally  overlapping  descriptors  can  be  chosen  by 
different  methods  and  with  different  criteria  that  depend  upon  the  purpose 
of  the  questionnaire,  the  theoretical  orientation  of  the  investigator,  the 
statistical  methods  employed,  etc.  Two  illustrative  procedures  are  de- 
scribed here  in  connection  with  Tables  7 and  8,  which  present  two  differ- 
ently scaled  lists  of  descriptors  denoting  degrees  of  acceptability. 

Consider  Table  7.  This  table  was  constructed  on  the  basis  of  the  in- 
formation presented  in  Table  2,  using  the  following  procedure.  First,  no 
candidate  from  Table  2 was  considered  if  its  standard  deviation  was  1 .00 
or  greater.  This  arbitrary  restriction  limited  the  candidates  to  those 
occupying  relatively  narrow  bands  on  the  favorableness  dimension  alorg 
which  they  were  evaluated.  Second,  from  the  remaining  candidates,  the  one 
having  the  largest  mean  (i.e.,  wholly  acceptable)  was  chosen  for  the  posi- 
tive extreme  of  the  response  dimension.  The  rest  of  the  descriptors  were 
then  selected  in  descending  order  so  that  the  interval  between  each  pair 
of  adjacent  means  was  greater  than  zero  but  as  small  as  possible  without 
being  less  than  the  larger  standard  deviation  of  the  pair.  This  rule 
guaranteed  that  the  descriptors  would  be  at  least  one  standard  deviation 
apart  in  all  cases. 

Table  8 portrays  another  list  constructed  from  the  candidate  descrip- 
tors in  Table  2.  In  this  case,  however,  the  midpoint  (i.e.,  neutral)  was 
chosen  as  the  starting  place,  and  the  selection  process  proceeded  in  both 


Table  6 


RESULTS  PERTAINING  TO  INTRASUBJECT  RESPONSE  CONSISTENCY 


Within 

1 ist 

Across  list 

Descriptor 

1st 

2nd  "Acceptability 

1 "Adequacy" 

"Goodness" 

usage 

usage 

Mean  responses 

Mildly 

acceptable 

Highly 

1 .80 

1.69 

-- 

-- 

unacceptable 

-4.22 

-4.29 

— 

Exceptional ly 

inadequate 

-4.68 

-4.56 

-- 

1 

Neutral 



.00 

.00 

.00 

' Borderline 

-- 

.00 

-.02 

-.06 

Marginal 

-.12 

— 

-.18 

Standard  deviations 

Mildly 

acceptable 

Highly 

.95 

.70 

-- 

-- 

unacceptable 

.58 

.54 

— 

-- 

Exceptional ly 

inadequate 

.51 

.64 

-- 

-- 

Neutral 

_ _ 

.00 

.00 

.00 

Borderl ine 

-- 

.20 

.32 

.31 

Marginal 

— 

.52 

-- 

.92 
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Table  7 


EXAMPLE  SET  OF  MINIMALLY  OVERLAPPING  DESCRIPTORS 
DENOTING  DEGREES  OF  "ACCEPTABILITY" 


Descriptor 

Mean 

SD 

Wholly  acceptable 

4.73 

.56 

Highly  acceptable 

4.04 

.63 

Reasonably  acceptable 

2.29 

.72 

Barely  acceptable 

1 .08 

.52 

Neutral 

.00 

.00 

Barely  unacceptable 

-1.00 

.30 

Somewhat  unacceptable 

-1 .77 

.67 

Substantially  unacceptable 

-3.24 

.90 

Highly  unacceptable 

-4.22 

.58 

Completely  unacceptable 

-4.90 

.36 

Note.  This  table  is  based  on  Table  2. 


Table  8 


ANOTHER  EXAMPLE  SET  OF  MINIMALLY  OVERLAPPING  DESCRIPTORS 
DENOTING  DEGREES  OF  "ACCEPTABILITY" 


Descriptor  Mean  SD 


Very  very  acceptable 

4.16 

.83 

Largely  acceptable 

3.14 

.99 

Mildly  acceptable 

1.69 

.70 

Sort  of  acceptable 

.94 

.65 

Neutral 

.00 

.00 

Barely  unacceptable 

-1.10 

.30 

Rather  unacceptable 

-2.02 

.84 

Substantially  unacceptable 

-3.24 

.90 

Highly  unacceptable 

-4.29 

.54 

Completely  unacceptable 

-4.90 

.36 

Note.  This  table  is  based  on  Table  2. 
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directions  from  the  middle.  The  fact  that  the  lower  halves  of  Tables  7 
and  8 are  identical  results  from  the  fortuitous  fact  that  neutral  oc- 
curred as  the  midpoint  in  Table  7. 

Tables  9 and  10,  constructed  in  the  same  manner  as  Table  7,  portray 
sets  of  descriptors  denoting,  respectively,  degrees  of  adequacy  and  degrees 
of  relative  goodness.  The  norms  in  these  tables  were  obtained  from  Tables 
3 and  4,  respectively. 


DISCUSSION  AND  CONCLUSIONS 

The  return  rate  of  92%  on  the  first  administration  of  the  test  lists 
was  reduced  to  54%  after  the  readministration.  Other  attrition  reduced  the 
final  number  of  usable  questionnaires  to  51%  of  the  original  total.  The 
precise  effect  of  this  loss  on  the  normative  data  presented  in  this  report 
is,  of  course,  unknown.  Some  amount  of  reduction  in  the  variability  of  re- 
sponses to  each  item  would  be  expected;  but  the  item  means  may  have  been 
largely  unaffected,  since  no  systematic  relevant  bias  due  to  the  attrition 
appears  obviously  noteworthy. 

The  number  of  descriptors  that  were  considered  ambiguous  was  quite 
small,  as  was  the  number  of  respondents  who  considered  any  descriptor  am- 
biguous. Therefore,  the  descriptors  listed  in  Table  1 are  not  necessarily 
unsuitable  for  use  as  scale  anchors  or  response  alternatives.  Neverthe- 
less, it  may  be  wise  to  exercise  discretion  in  using  the  modifiers  pretty 
and  rather,  which  occurred  in  more  than  half  the  nonmidpoint  descriptors 
in  the  table.  Three  of  the  descriptors  in  Table  1 — mildly  adequate,  con- 
siderably inadequate,  and  neutral — were  used  in  the  example  sets  given  in 
Tables  7,  8,  and  9.  However,  those  three  tables  (along  with  Table  10) 
were  intended  to  be  not  so  much  examples  of  universally  ideal  lists  of  de- 
scriptors as  illustrations  of  possible  methods  for  constructing  good  ques- 
tionnaire items. 

As  regards  descriptors  for  scale  midpoints,  the  term  neutral  appears, 
at  first  glance,  to  be  ideal:  both  its  mean  and  standard  deviation  were 
zero  on  all  three  test  lists.  However,  in  a study  described  by  Gividen 
(1973),  subjects  (Army  test  officers)  displayed  some  confusion  about  the 
precise  meaning  of  the  term.  Some  interpreted  it  as  denoting  no  opinion 
rather  than  as  an  opinion  halfway  between  scale  extremes.  Furthermore,  a 
majority  of  the  subjects  held  the  opinion  that  because  of  the  ambiguity  in 
the  term  it  should  not  be  used  in  questionnaires.  Thus,  although  the  mean 
and  standard  deviations  observed  in  the  present  study  favor  the  use  of 
neutral  as  a scale  midpoint,  in  practical  situations  its  use  may  sometimes 
make  the  interpretation  of  obtained  data  difficult  or  impossible.  Conse- 
quently, neutral  should  be  employed  only  after  prior  consideration  indi- 
cates that  confusion  would  not  exist  within  the  particular  context. 

According  to  Gividen  ( 1973),  the  term  borderline  does  not  suffer  the 
potential  disadvantage  of  ambiguity  associated  with  neutral . In  the  pres- 
ent study,  borderline  was  reported  as  confusing  by  4 out  of  the  51  subjects; 
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EXAMPLE  SET  OF  MINIMALLY  OVERLAPPING  DESCRIPTORS 
DENOTING  DEGREES  OF  "ADEQUACY" 


Descriptor 

Mean 

SD 

Totally  adequate 

4.62 

.85 

Very  adequate 

3.42 

.85 

Reasonably  adequate 

2.41 

.77 

Mildly  adequate 

1 .57 

.67 

Barely  adequate 

.63 

.93 

Barely  inadequate 

-1.16 

.64 

Somewhat  inadequate 

-1 .88 

.73 

Considerably  inadequate 

-3.60 

.68 

Very  very  inadequate 

-4.46 

.54 

Note.  This  table  is  based  on  Table  3. 


Table  10 

EXAMPLE  SET  OF  MINIMALLY  OVERLAPPING  DESCRIPTORS 
DENOTING  DEGREES  OF  "RELATIVE  GOODNESS" 


Descriptor 

Mean 

SD 

Best  of  all 

4.90 

.51 

Extremely  better 

3.92 

.88 

Moderately  better 

2.26 

.74 

Slightly  better 

1.16 

.78 

Alike 

.22 

.85 

Barely  worse 

-1.04 

.82 

Somewhat  worse 

-2.08 

.86 

Conspicuously  worse 

-3.28 

.89 

Absolutely  worse 

-4.43 

.82 

Note.  This  table  is  based  on  Table  4. 


neutral  by  2 . Regardless,  it  does  not  seem  likely  that  borderline  would 
be  used  by  subjects  to  indicate  no  opinion.  This  conclusion,  coupled  with 
the  fact  that  the  means  and  standard  deviations  observed  in  the  present 
study  for  borderline  were  close  to  zero,  suggests  that  the  term  is  gener- 
ally appropriate  as  a midpoint  descriptor.  In  contrast,  neutral  may  be 
normally  inappropriate. 

An  examination  of  Tables  2,  3,  and  4 revealed  that  the  descriptors 
with  the  greatest  amount  of  item  variability  tended  to  be  characterized 
by  extremeness  or  generality  (see  Table  5).  Why  descriptors  such  as  best, 
absolutely  worst,  and  the  like  should  be  interpreted  so  differently  by  dif 
ferent  respondents  is  unclear.  It  is  interesting  to  speculate  that  there 
is  considerable  variability  among  subjects  in  the  tendency  to  assign  ex- 
treme values.  As  to  terms  like  acceptable  and  unacceptable,  it  may  be 
that  such  unmodified  common  terms  of  moderate  degree  possess  relatively 
little  specificity  of  meaning.  The  variability  of  the  terms  in  Table  5 
would  perhaps  shrink  to  some  extent  if  the  terms  were  located  among  other 
terms  in  actual  questionnaire  items. 

The  two  methods  for  selecting  descriptors,  which  were  illustrated  in 
Tables  7,  8,  9,  and  10,  are  easily  subjected  to  reasonable  variations. 

One,  for  example,  would  be  to  begin  by  choosing  from  the  candidate  pool 
the  two  descriptors  that  have  the  highest  and  lowest  means  and  the  descrip 
tor  with  the  mean  closest  to  zero.  Low-variability  intermediate  descrip- 
tors would  then  be  selected  from  the  rest  of  the  pool  in  a manner  similar 
to  that  used  for  Tables  7,  8,  9,  and  10. 

Note  that  the  scale  values  (means)  obtained  in  this  study  should  not 
be  used  as  scale  values  in  the  analysis  of  questionnaire  items.  Rather, 
they  should  be  used  only  as  aids  in  selecting  response  alternatives  or 
scale  anchors  for  questionnaire  items. 

Note  also  that  the  descriptor  sets  presented  in  Tables  7,  8,  9,  and 
10  were  derived  using  criteria  related  only  to  scale  position  and  item 
variability.  There  may  be  other  important  considerations  related  to  the 
selection  of  response  alternatives  or  scale  anchors  for  any  given  real 
questionnaire.  Many  of  these  potential  considerations  are  discussed  in 
the  ARI  Questionnaire  Construction  Manual  (Dyer  et  al.  , 1976).  That  rran- 
ual  also  presents  sample  lists  of  descriptors  selected  with  regard  to 
other  criteria. 
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24 


